System and method for improving lidar data fidelity using pixel-aligned lidar/electro-optic data

ABSTRACT

A lidar and digital camera system collect data and generate a three-dimensional image. A lidar generates a laser beam to form a lidar shot and to receive a reflected laser beam to provide range data. A digital camera includes an array of pixels to receive optical radiation and provide electro-optical data. An optical bench passes the laser beam, reflected laser beam, and optical radiation and is positioned to align each pixel to known positions within the lidar shot. Pixels are matched to form a lidar point-cloud which is used to generate an image.

RELATED APPLICATIONS

This application claims the benefit of and is a divisional of co-pendingU.S. patent application Ser. No. 11/244,212 filed on Oct. 5, 2005.

TECHNICAL FIELD

The present invention relates to three-dimensional modeling. Morespecifically, the present invention relates to a system and method thatincorporates a lidar to perform three-dimensional imagery in real time.

BACKGROUND OF THE INVENTION

Lidar (light detection and ranging) uses laser technology to makeprecise distance measurements over long or short distances. Oneapplication of lidar is the range scanner, or scanning lidar. Lidartransceivers operate on the principle of transmitting laser light thatthen reflects off of a given object and returns to a lidar receiver. Thedistance to an object is then determined by analyzing the laser signalthrough various techniques. During the scanning process, the lidar makesrepeated range measurements to objects in its path. Through repeatedmeasurements of an object by individual laser transmissions, the shapeof the object can be determined. The resulting range data is collectedand serves as a rough model of the scanned area.

In many applications, obtaining high-resolution, high-fidelity shapeinformation is desirable. Physical limitations of the range scannerconstrain the maximum spatial resolution of the range data, whichdecreases with distance from the range scanner. At large distances, therange scanner may not be able to discern surface details of an object.Thus, it would be an advancement in the art to improve the ability toachieve an increased resolution of shape information obtained by lidartransceivers from long distances.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are now described with reference tothe figures, in which:

FIG. 1 is a block diagram illustrating a system for collecting lidar andelectro-optical data.

FIG. 2 is a diagram illustrating focal planes of a system for collectinglidar and electro-optical data.

FIG. 3 is a diagram illustrating a lidar shot on an object.

FIG. 4 is a diagram illustrating multiple lidar shots.

FIG. 5 is a diagram illustrating multiple lidar point-clouds.

FIG. 6 is a flow diagram illustrating a technique for collecting lidarand electro-optical data to generate an image.

DETAILED DESCRIPTION

The presently preferred embodiments of the present invention will bebest understood by reference to the drawings, wherein like parts aredesignated by like numerals throughout. It will be readily understoodthat the components of the present invention, as generally described andillustrated in the figures herein, could be arranged and designed in awide variety of different configurations. Thus, the following moredetailed description of the embodiments of the apparatus, system, andmethod of the present invention, as represented in FIGS. 1 through 6, isnot intended to limit the scope of the invention, as claimed, but ismerely representative of presently preferred embodiments of theinvention.

Referring to FIG. 1, a block diagram of an image capture system 100 isshown. The system includes a lidar 102 for scanning an object 104 togenerate range data, i.e., distance measurements from the lidar 102 toreal-world objects. The object 104 may be any indoor or outdoorthree-dimensional region to which distance measurements can be madeusing the lidar 102. By way of example, the lidar 102 may be embodied asan LMS 291, manufactured by SICK AG of Waldkirch, Germany, althoughvarious other models are contemplated. The lidar 102 includes a lidartransmitter 106 to transmit laser radiation and a lidar receiver 108 tocapture and convert received laser radiation.

The system 100 includes a high-resolution, high-speed digital camera 110for obtaining digital images of the object 104 during an imagingprocess. The digital camera 110 may include an ISG LightWiseLW-3-S-1394™. camera manufactured by Imaging Solutions Group of NY. Thedigital camera 110 includes an electro-optic (EO) detector array 111that generates EO pixels for a field-of-view. The EO detector array 111captures various forms of active and passive optical radiation.

The system 100 includes an optical bench 112 to pass and directradiation. The optical bench 112 may include a plurality of optics thatoperate specific to the lidar 102 and the digital camera 110.

The system 100 includes a controller 114 that directs operation of thesystem components. The controller 114 may be embodied as amicroprocessor, microcontroller, digital signal processor (DSP), orother control device known in the art. The controller 114 is coupled toa memory 116 that may include a random access memory (RAM) 118 and aread only memory (ROM) 120 or the like. The memory 116 buffers rangedata and digital images during the imaging process. The memory 116 mayalso be used to store parameters and program code for system operation.

The controller 114 is in electrical communication with a controlinterface 122 to enable user interaction. The control interface 122 maybe implemented as a universal serial bus (USB) interface, RS-232interface, or wireless interface, such as an 802.11b interface orinfrared (IR) receiver. In one embodiment, the controller 114 mayfurther be in communication with a communication interface 124 fortransmitting captured range data, digital images, and 3D images. Thecommunication interface 124 may include, for instance, an Ethernetadapter, a IEEE 1349 (Firewire) adaptor, a USB adaptor, or otherhigh-speed communication interface.

In operation, the controller 114 instructs the lidar transmitter 106 totransmit laser radiation in the form of a laser beam 126 to the opticalbench 112. The optical bench 112 directs the laser beam 126 to aspecific solid angle within a field-of-view (FOV) of the lidar receiver108 and digital camera 110. The directed laser beam 126 reflects off theobject 104 and reflected laser radiation 128 returns to the opticalbench 112. The optical bench 112 captures the reflected laser radiation128 from the same solid angle of the FOV. The optical bench 112 directsthe reflected laser radiation 128 to the lidar receiver 108 whichconverts the captured laser radiation 128 to electrical signals. Theelectrical signals are transmitted to the controller 114 that computesrange data. The range data is based on the distance that the laser lighttraveled using techniques known to those skilled in the art, such asheterodyne or time-of-flight techniques.

At the same time the laser radiation is transmitted, the controller 114instructs the digital camera 110 to collect optical radiation 130 fromthe same solid angle of the FOV. The solid angle within the FOV of thedigital camera 110 encompasses the same solid angle within the FOV asthe lidar transmitter 106 and the lidar receiver 108. The optical bench112 captures the optical radiation 130 from the solid angle of the FOV.The optical bench 112 directs the captured optical radiation 130 to thedigital camera 110. Once captured, the digital camera 110 converts theoptical radiation 130 to electronic spectral data that is sent to thecontroller 114 for processing.

The controller 114 repeatedly commands the components to initiate theabove-described sequence in a manner such that the range data andelectronic spectral data are generated in sequence. Range data andelectronic spectral data are assembled together by an image match module131 to construct an image. In this manner, the controller 114sequentially builds a spectrally textured range image. The position of aspectrally textured range image in space is known relative to theoptical axis of the optical bench 112. The controller 114 rectifies therelative position and orientation of a spectrally textured range imageto a relative position and orientation within a local coordinate system.

The system 100 may include a position and orientation system 132 togenerate position and orientation data with respect to the Earth. Theposition and orientation system 132 may provide a bearing or heading(azimuth) of the system 100. Azimuth is typically expressed as ahorizontal angle of the observers bearing, measured clockwise from areferent direction, such as North. A position and orientation system 132may include a high-accuracy compass capable of digital output. Incertain implementations, the position and orientation system 132 mayprovide the tilt or inclination of the lidar 102 with respect to theEarth's surface. For example, the lidar 102 may be tilted with respectto one or two axes. For simplicity, however, the following exemplaryembodiments assume that the lidar 102 is level prior to scanning.

The controller 114 receives the position and orientation data withrespect to the Earth and rectifies the relative position and orientationof the spectrally textured range image to a global coordinate system. Aclock 134 provides a timing signal to the controller 114 and theposition and orientation system 132, to ensure the system position andorientation is known at the exact time of the data acquisition.

Following the rectification, the controller 114 converts a spectrallytextured range image into a 3D image patch. The 3D image patch may bestored in a mass storage device 136, such as a hard disk drive, opticalstorage device (e.g., DVD-RW) memory. The mass storage device 136 may beconstrued to include any combination of volatile, non-volatile,magnetic, or optical storage media.

An object shape is represented by a multitude of 3D points derived fromthe processing of lidar signals. The 3D points are collectively referredto herein as a point-cloud. Techniques and implementations disclosedherein improve the ability to obtain high-fidelity, high-resolutionlidar point-clouds without the use of improved lidar technology orimproved platform stabilization technology.

Referring to FIG. 2, a diagram illustrates one optical design 200 forthe system 100. As can be appreciated, numerous optical designs may beimplemented, and the illustration of FIG. 2 is for illustrative purposesonly. The optical design 200 provides both a lidar focal plane 202 andan EO focal plane 204. An optic 206 may be incorporated, such as withinthe optical bench 112, to direct captured radiation to the respectiveplanes 202, 204. The EO focal plane 204 detects any form of active orpassive optical radiation collected on detectors positioned on a focalplane. In the system 100, the EO focal plane 204 coincides with thedigital camera 110, and the lidar focal plane 202 coincides with thelidar receiver 108.

The optical bench 112 aligns the focal planes 202, 204 such that thelidar 102 and the digital camera 110 are aligned as long as theirsubtended angles are within the FOV. Accordingly, the alignment of alaser beam 126 is automatically matched or aligned with the opticalradiation 130 sensed by the digital camera 110 that generates EO pixelsfor the same FOV. Thus, if a transmitted laser beam 126 casts afootprint on an object in the FOV, the corresponding pixels on the EOfocal plane 204 that are directed at the same object are immediately andunambiguously known. The lidar footprint and the EO pixels are thusaligned at the pixel level (pixel-aligned). The EO detector array 111and the lidar receiver 108 generate image data that is read from thesame portion of the FOV at the same time.

Referring to FIG. 3, a layout 300 illustrates a FOV 302 of a system 100relative to an object 104. As can be appreciated, the object 104 may beentirely or partially encompassed within a FOV 302. A single lidar shot304 that casts a footprint upon the object 104 is illustrated. The EOdetector array 111 is designed such that a single lidar shot 304corresponds to a plurality of aligned EO pixels 306. The controller 114identifies each EO pixel 306 falling within the lidar shot 304. Thesystem 100 may be designed such that dozens or hundreds of EO pixels 306of the array 111 fall within the single lidar shot 304. This enables thedraping of detailed spectral information onto a relatively coarsesurface shape defined by the aligned but more broadly spaced 3D pointsmeasured by the lidar 102. The pixel-alignment of an EO detector array111 with a lidar 102 allows for generation of very high resolutionpoint-cloud. As can be expected, multiple lidar shots 304, eachincluding a plurality of EO pixels 306, are required to properly imagean object 104.

Referring to FIG. 4, a layout 400 illustrates a plurality of lidar shots402 that may be taken in rapid succession. A plurality or array of lidarshots 402 may be acquired through the use of a lidar 102 or an array oflidars 102. Each lidar shot 402 is representative of a 3D point, andeach lidar shot 402 includes a plurality of aligned EO pixels 404. TheEO pixels 404 have a higher spatial resolution than the lidar shot 402.The lidar shots 402 may include overlap areas 406 relative to oneanother. The image match module 131 identifies pixels 404 within overlapareas 406 and arranges lidar shots 402 relative to one another. Asufficient number of 3D lidar shots 402 may be arranged together to forma lidar point-cloud.

Referring to FIG. 5, a series of lidar point-clouds 500 a-c are shown.Each lidar point-cloud 500 a-c includes a plurality of lidar shots 502,with each lidar shot 502 including a plurality of aligned EO pixels. Thelidar point-clouds 500 a-c illustrate different views of an object andmay be merged to realize the entire object.

A resolution of a first point-cloud may be coarse, but as more lidarshots are taken, the image increases in resolution. Data collection maybe repeated to obtain a second set of data for the same or partiallyoverlapping area within the FOV. The second set of data may be from aslightly different view point. Thus, two point-clouds may be for thesame or partially overlapping area. By combining coarser images given bypoint-clouds, a finer resolution is realized. By overlayingpoint-clouds, fewer views may be required to receive the same detail.

If the change in view point is significant, the controller 114 mergestwo partially overlapping point-clouds using the image match module 131that operates in real time to provide 2D image matching. The image matchmodule 131 may match overlapping point-clouds by EO pixel-to-pixelcomparison.

Given the image matching results, the image match module 131 generates apointing error. Based on the pointing error, the controller 114 appliescorrections for errors in position, orientation, and/or time thatoccurred between the two data collections. Following the pointing errorcorrection, platform position and orientation data along with sensorpointing data are used to combine two point-clouds into one higherresolution point-cloud.

Data collection and image matching are repeated to add additionalpoint-clouds. The additional point-clouds are aligned with the previouspoint-clouds. As long as the system 100 can continue pointing at andcollecting lidar data from an object 104, a point-cloud may be refined.

The accuracy of the merging of multiple point-clouds is directlycorrelated with the resolution of the EO images being used and theaccuracy and robustness of the image match module 131. For example, asystem 100 may be designed with a lidar 102 having a 100 microradianspatial resolution, and a digital camera 110 with pixels that subtend 10microradians in each dimension. If the 10 microradian pixels for twosuccessive images are matched to an accuracy of about one pixel, theresulting 10 or so microradians of relative pointing data may then beused to correct the pointing error of the system and thereby moreaccurately merge two point-clouds. An original point-cloud density of100 shots per square milliradians (100 shots per square meter at adistance of 1000 meters) could be effectively improved to 200 shots persquare milliradians. Even more shots per square milliradians may then beaccumulated through the merging and accumulation of subsequentpoint-cloud datasets.

Referring to FIG. 6, a flow diagram 600 illustrates an embodiment of aprocess performed by the present invention. A lidar, or an array oflidars, collects 602 a plurality of 3D lidar shots which form a lidarpoint-cloud. Approximately simultaneously to the lidar point-cloud datacollection, a digital camera collects 604 an EO image from opticalradiation. The EO image typically has a higher resolution than the lidarpoint-cloud. The EO image includes multiple pixels, with each pixelaligned with the point-cloud data. The data collection for the lidarpoint-cloud and the EO image is preferably completed in sufficient timeso as to not suffer the effects of object or scene motion within theFOV.

Data collection in steps 602 and 604 may be repeated to obtain one ormore additional data sets. An additional data set may correspond to thesame area as a first data set and, therefore, be an overlaying lidarpoint-cloud. An additional data set for a lidar point-cloud may alsocorrespond to an overlapping area. By taking multiple data sets of thesame area, an initially course image may be refined to provide an imagewith high resolution. Thus, by use of a system with a high resolutiondigital camera and a low resolution lidar, a high resolution lidar dataset and combined image is achieved. Such a system is less expensive andmore reliable than one incorporating a high resolution lidar.

Multiple lidar shots may partially or fully overlap one another. Theimage match module 131 identifies and matches 606 lidar shots relativeto one another. Once the lidar shots are matched, the image match module131 is able to merge 608 lidar shots relative to one another. The imagematch module 131 matches 610 a data set with a previous data set. Thedata set and previous data set are then merged 612 together.

After the imaging matching and merging, the controller 114 generates 614a pointing error and corrects that error that occurred between datacollection times. The controller 114 then merges 616 all data sets togenerate a final image. Overlapping data sets are thereby merged into asingle higher resolution lidar point-cloud. Overlapping data sets aremerged into a single lidar point-cloud to increase the point cloud andimage size and increase resolution for the overlap areas. Additionaldata sets collected from slightly different viewpoints are effective infurther refining a point cloud and an image.

Data collection and merging may be repeated for additional data sets tobuild and refine an image. A system may continue collecting data as longas a target of interest remains within the FOV. As an object is refinedover multiple data set collections, data collection may commence from agreater distance. By overlaying and merging data collections, a highresolution image is developed with multiple “course” data collectionsrather than one fine image.

While specific embodiments and applications of the present inventionhave been illustrated and described, it is to be understood that theinvention is not limited to the precise configuration and componentsdisclosed herein. Various modifications, changes, and variationsapparent to those of skill in the art may be made in the arrangement,operation, and details of the methods and systems of the presentinvention disclosed herein without departing from the spirit and scopeof the present invention.

1. A method for generating a three-dimensional image, comprising:aligning pixels in an electro-optical array with a lidar such that thepixels are positioned within a lidar shot formed by the lidar; acquiringa plurality of lidar shots, wherein each lidar shot is associated with aplurality of aligned pixels and is acquired by, the lidar transmitting alaser beam to form a lidar shot, the lidar receiving a reflected laserbeam indicative of range data, and the electro-optical array receivingoptical radiation indicative of electro-optical data, wherein theelectro-optical data corresponds to aligned pixels within the lidarshot; arranging the plurality of lidar shots relative to one anotherusing the aligned pixels associated with the plurality of lidar shots;forming a first point-cloud from the arranged plurality of lidar shots;merging the first point-cloud with a second point-cloud, the secondpoint-cloud comprising a second plurality of lidar shots, eachassociated with a plurality of aligned pixels, and wherein the firstpoint-cloud is merged with the second point-cloud using the alignedpixels associated with the lidar shots of the first point-cloud and thealigned pixels associated with the lidar shots of the secondpoint-cloud.
 2. A method for generating a three-dimensional image,comprising: aligning pixels in an electro-optical array with a lidarsuch that the pixels are positioned within a lidar shot formed by thelidar; acquiring a plurality of lidar shots, wherein each lidar shot isassociated with a plurality of aligned pixels and is acquired by, thelidar transmitting a laser beam to form a lidar shot, the lidarreceiving a reflected laser beam indicative of range data, and theelectro-optical array receiving optical radiation indicative ofelectro-optical data, wherein the electro-optical data corresponds toaligned pixels within the lidar shot; arranging the plurality of lidarshots relative to one another using the aligned pixels associated withthe plurality of lidar shots; forming a first point-cloud from thearranged plurality of lidar shots; forming a second point-cloud from thearranged plurality of lidar shots; wherein the first point-cloud isassociated with first lidar position data and the second point-cloud isassociated with second lidar position data, and wherein merging thefirst and second point-clouds comprises arranging the first point-cloudrelative to the second point-cloud using the lidar position dataassociated with, the first and the second point-clouds.
 3. The method ofclaim 2, wherein a footprint of one or more of the lidar shots of thefirst point-cloud overlap an area within a lidar footprint of one ormore of the lidar shots of the second point-cloud, and wherein mergingthe first and the second point-clouds comprises refining the arrangementof the first point-cloud relative to the second point-cloud by imagematching within the one or more overlap areas.
 4. The method of claim 2,further comprising calculating a pointing error in the first and thesecond lidar position data using the image matching.
 5. The method ofclaim 4, wherein the lidar position data comprises lidar position, lidarorientation, and lidar shot timing data.